1. Field of the Invention
The present invention relates generally to the automated assessment of abnormalities in images, and more particularly to methods, systems, and computer program products for computer-aided detection of abnormalities (such as lesions and lung nodules) in medical images (such as low-dose CT scans) using artificial intelligence techniques, including massive training artificial neural networks, (MTANNs).
The present invention also generally relates to computerized techniques for automated analysis of digital images, for example, as disclosed in one or more of U.S. Pat. Nos. 4,839,807; 4,841,555; 4,851,984; 4,875,165; 4,907,156; 4,918,534; 5,072,384; 5,133,020; 5,150,292; 5,224,177; 5,289,374; 5,319,549; 5,343,390; 5,359,513; 5,452,367; 5,463,548; 5,491,627; 5,537,485; 5,598,481; 5,622,171; 5,638,458; 5,657,362; 5,666,434; 5,673,332; 5,668,888; 5,732,697; 5,740,268; 5,790,690; 5,832,103; 5,873,824; 5,881,124; 5,931,780; 5,974,165; 5,982,915; 5,984,870; 5,987,345; 6,011,862; 6,058,322; 6,067,373; 6,075,878; 6,078,680; 6,088,473; 6,112,112; 6,138,045; 6,141,437; 6,185,320; 6,205,348; 6,240,201; 6,282,305; 6,282,307; 6,317,617; as well as U.S. patent applications Ser. No. 08/173,935; 08/398,307 (PCT Publication WO 96/27846); 08/536,149; 08/900,189; 09/027,468; 09/141,535; 09/471,088; 09/692,218; 09/716,335; 09/759,333; 09/760,854; 09/773,636; 09/816,217; 09/830,562; 09/818,831; 09/842,860; 09/860,574; 60/160,790; 60/176,304; 60/329,322; 09/990,311; 09/990,310; 60/332,005; 60/331,995; and 60/354,523; as well as co-pending U.S. patent applications as well as PCT patent applications PCT/US98/15165; PCT/US98/24933; PCT/US99/03287; PCT/US00/41299; PCT/US01/00680; PCT/US01/01478 and PCT/US01/01479, all of which are incorporated herein by reference.
The present invention includes use of various technologies referenced and described in the above-noted U.S. Patents and Applications, as well as described in the documents identified in the following LIST OF REFERENCES, which are cited throughout the specification by the corresponding reference number in brackets:
1. M. Kaneko, K. Eguchi, H. Ohmatsu, R. Kakinuma, T. Naruke, K. Suemasu, and N. Moriyama, xe2x80x9cPeripheral lung cancer: Screening and detection with low-dose spiral CT versus radiography,xe2x80x9d Radiology, vol. 201, pp. 798-802 (1996).
2. S. Sone, S. Takashima, F. Li, et al., xe2x80x9cMass screening for lung cancer with mobile spiral computed topography scanner,xe2x80x9d The Lancet, vol. 351, pp. 1242-1245 (1998).
3. C. I. Henschke, D. I. McCauley, D. F. Yankelevitz, et al., xe2x80x9cEarly Lung Cancer Action Project: Overall design and findings from baseline screening,xe2x80x9d The Lancet, vol. 354, pp. 99-105 (1999).
4. J. W. Gurney, xe2x80x9cMissed lung cancer at CT: Imaging findings in nine patients,xe2x80x9d Radiology, vol. 199, pp. 117-122 (1996).
5. F. Li, S. Sone, H. Abe, H. MacMahon, S. G. Armato III, and K. Doi, xe2x80x9cLung cancers missed at low-dose helical CT screening in a general population: Comparison of clinical, histopathologic, and image findings,xe2x80x9d Radiology, vol. 225, pp. 673-683 (2002).
6. S. Yamamoto, I. Tanaka, M. Senda, Y. Tateno, T. linuma, T. Matsumoto, and M. Matsumoto, xe2x80x9cImage processing for computer-aided diagnosis of lung cancer by CT (LDCT),xe2x80x9d Systems and Computers in Japan, vol. 25, pp. 67-80 (1994).
7. T. Okumura, T. Miwa, J. Kako, S. Yamamoto, M. Matsumoto, Y. Tateno, T. linuma, and T. Matsumoto, xe2x80x9cImage processing for computer-aided diagnosis of lung cancer screening system by CT (LDCT),xe2x80x9d Proc. SPIE, vol. 3338, pp. 1314-1322 (1998).
8. W. J. Ryan, J. E. Reed, S. J. Swensen, and J. P. F. Sheedy, xe2x80x9cAutomatic detection of pulmonary nodules in CT,xe2x80x9d Proc. of Computer Assisted Radiology, pp. 385-389 (1996).
9. K. Kanazawa, M. Kubo, N. Niki, H. Satoh, H. Ohmatsu, K. Eguchi, and N. Moriyama, xe2x80x9cComputer assisted lung cancer diagnosis based on helical images,xe2x80x9d Image Analysis Applications and Computer Graphics: Proc. of Int. Computer Science Conf., pp. 323-330 (1995).
10. M. L. Giger, K. T. Bae, and H. MacMahon, xe2x80x9cComputerized detection of pulmonary nodules in computed tomography images,xe2x80x9d Investigative Radiology, vol. 29, pp. 459-465 (1994).
11. S. G. Armato III, M. L. Giger, J. T. Blackbur, K. Doi, and H. MacMahon, xe2x80x9cThree-dimensional approach to lung nodule detection in helical CT,xe2x80x9d Proc. of SPIE, vol. 3661, pp. 553-559 (1999).
12. S. G. Armato III, M. L. Giger, C. J. Moran, J. T. Blackbur, K. Doi, and H. MacMahon, xe2x80x9cComputerized detection of pulmonary nodules on CT scans,xe2x80x9d Radiographics, vol. 19, pp. 1303-1311 (1999).
13. S. G. Armato III, M. L. Giger, and H. MacMahon, xe2x80x9cAutomated detection of lung nodules in CT scans: Preliminary results,xe2x80x9d Medical Physics, vol. 28, pp. 1552-1561 (2001).
14. S. G. Armato III, F. Li, M. L. Giger, H. MacMahon, S. Sone, and K. Doi, xe2x80x9cLung cancer: Performance of automated lung nodule detection applied to cancers missed in a CT screening program,xe2x80x9d Radiology, vol. 225, pp. 685-692 (2002).
15. K. Suzuki, S. G. Armato III, F. Li, S. Sone, and K. Doi, xe2x80x9cMassive training artificial neural network (MTANN) for reduction of false positives in computerized detection of lung nodules in low-dose CT,xe2x80x9d (Submitted to) Medical Physics, (2003).
16. K. Suzuki, I. Horiba, N. Sugie, and M. Nanki, xe2x80x9cNoise reduction of medical X-ray image sequences using a neural filter with spatiotemporal inputs,xe2x80x9d Proc. Int. Symp. Noise Reduction for Imaging and Communication Systems, pp. 85-90 (1998).
17. K. Suzuki, I. Horiba, and N. Sugie, xe2x80x9cTraining under achievement quotient criterion,xe2x80x9d IEEE Neural Networksfor Signal Processing X, pp. 537-546 (2000).
18. K. Suzuki, I. Horiba, and N. Sugie, xe2x80x9cSignal-preserving training for neural networks for signal processing,xe2x80x9d Proc. of IEEE Int. Symp. Intelligent Signal Processing and Communication Systems, vol. 1, pp. 292-297 (2000).
19. K. Suzuki, I. Horiba, and N. Sugie, xe2x80x9cDesigning the optimal structure of a neural Filter,xe2x80x9d IEEE Neural Networksfor Signal Processing VIII, pp. 323-332 (1998).
20. K. Suzuki, I. Horiba, and N. Sugie, xe2x80x9cA simple neural network pruning algorithm with application to filter synthesis,xe2x80x9d Neural Processing Letters, vol. 13, pp. 43-53 (2001).
21. K. Suzuki, I. Horiba, and N. Sugie, xe2x80x9cSimple unit-pruning with gain-changing training,xe2x80x9d IEEE Neural Networks for Signal Processing XI, pp. 153-162 (2001).
22. K. Suzuki, I. Horiba, and N. Sugie, xe2x80x9cEfficient approximation of a neural filter for quantum noise removal in X-ray images,xe2x80x9d IEEE Transactions on Signal Processing, vol. 50, pp. 1787-1799 (2002).
23. K. Suzuki, I. Horiba, N. Sugie, and M. Nanki, xe2x80x9cNeural filter with selection of input features and its application to image quality improvement of medical image sequences,xe2x80x9d IEICE Transactions on Information and Systems, vol. E85-D, pp. 1710-1718 (2002).
24. K. Suzuki, I. Horiba, and N. Sugie, xe2x80x9cEdge detection from noisy images using a neural edge detector,xe2x80x9d IEEE Neural Networks for Signal Processing X, pp. 487-496 (2000)
25. K. Suzuki, I. Horiba, and N. Sugie, xe2x80x9cNeural edge detector -a good mimic of conventional one yet robuster against noise-,xe2x80x9d Lecture Notes in Computer Science, Bio-Inspired Applications of Connectionism, vol. 2085, pp. 303-310 (2001).
26. K. Suzuki, I. Horiba, N. Sugie, and M. Nanki, xe2x80x9cExtraction of the contours of left ventricular cavity, according with those traced by medical doctors, from left ventriculograms using a neural edge detector,xe2x80x9d Proc. of SPIE, vol. 4322, pp. 1284-1295 (2001).
27. K. Suzuki, I. Horiba, N. Sugie, and M. Nanki, xe2x80x9cContour extraction of the left ventricular cavity from digital subtraction angiograms using a neural edge detector,xe2x80x9d Systems and Computers in Japan, vol. 34, pp. 55-69 (2003).
28. K. Suzuki, I. Horiba, K. Ikegaya, and M. Nanki, xe2x80x9cRecognition of coronary arterial stenosis using neural network on DSA system,xe2x80x9d Systems and Computers in Japan, vol. 26, pp. 66-74 (1995).
29. K. Suzuki, I. Horiba, N. Sugie, and M. Nanki, xe2x80x9cComputer-aided diagnosis system for coronary artery stenosis using a neural network,xe2x80x9d Proc. of SPIE, vol. 4322, pp. 1771-1782 (2001).
30. D. E. Rumelhart, G. E. Hinton, and R. J. Williams, xe2x80x9cLearning representations of back-propagation errors,xe2x80x9d Nature, vol. 323, pp. 533-536 (1986).
31. D. E. Rumelhart, G. E. Hinton, and R. J. Williams, xe2x80x9cLearning internal representations by error propagation,xe2x80x9d vol. 1 of Parallel Distributed Processing, MIT Press, MA, pp. 318-362 (1986).
32. D. P. Chakraborty and L. H. L. Winter, xe2x80x9cFree-response methodology: Alternate analysis and a new observer-performance experiment,xe2x80x9d Radiology, vol. 174, pp. 873-881 (1990).
33. K. Funahashi, xe2x80x9cOn the approximate realization of continuous mappings by neural networks,xe2x80x9d Neural Networks, vol. 2, pp. 183-192 (1989).
34. A. R. Barron, xe2x80x9cUniversal approximation bounds for superpositions of a sigmoidal function,xe2x80x9d IEEE Transactions on Information Theory, vol. 39, pp. 930-945 (1993).
35. C. E. Metz, xe2x80x9cROC methodology in radiologic imaging,xe2x80x9d Invest. Radiology, vol. 21, pp. 720-733 (1986).
36. C. E. Metz, B. A. Herman, and J.-H. Shen, xe2x80x9cMaximum likelihood estimation of receiver operating characteristic (ROC) curves from continuously-distributed data,xe2x80x9d Stat. Med., vol. 17, pp. 1033-1053 (1998).
The contents of each of these references, including patents and patent applications, are incorporated herein by reference. The techniques disclosed in the patents, patent applications, and other references can be utilized as part of the present invention.
2. Discussion of the Background
Lung cancer continues to rank as the leading cause of cancer deaths among Americans. Screening programs for lung cancer have been carried out with low-dose helical CT (LDCT) [1-3] because early detection of lung cancer allows a more favorable prognosis for the patient. In lung cancer screening, radiologists must read many CT images, resulting possibly in missing some cancers during the interpretation [4][5]. Therefore, computer-aided diagnostic (CAD) schemes for lung nodule detection in LDCT has been investigated as a useful tool for lung cancer screening.
Many investigators have developed CAD schemes for lung nodule detection in CT based on morphological filtering [6][7], geometric modeling [8], fuzzy clustering [9], and gray-level thresholding [10-14]. A major problem with CAD schemes is the large number of false positives, which would cause difficulty in the clinical application of the CAD schemes. Therefore, it is important to reduce the number of false positives as much as possible, while maintaining high sensitivity. Some false-positive reduction techniques have been developed by use of a classifier, such as an artificial neural network (ANN). An ANN usually requires training with a large number of cases, e.g., 500 cases, to achieve adequate performance. If the ANN is trained with a small number of cases, the generalization ability (performance for non-training cases) is lower, i.e., the ANN fits only the training cases, which is known as xe2x80x9cover-training.xe2x80x9d
In the field of image processing, Suzuki et al. have developed a nonlinear filter based on a multilayer ANN called a xe2x80x9cneural filterxe2x80x9d [16-21] and applied it for reduction of the quantum mottle in X-ray images [22][23]. They developed a supervised edge detector based on a mutilayer ANN, called a xe2x80x9cneural edge detector,xe2x80x9d [24][25] and applied it for detection of
Since diagnostic radiology is progressing rapidly with associated technological advances, the timely development of CAD schemes for diagnostic radiology is very important. However, it is very difficult to collect a large number of abnormality training cases, particularly for a CAD scheme for diagnosis with a new modality, such as lung cancer screening with CT. Accordingly, it becomes very difficult to train CAD systems, e.g., artificial neural networks, in these new modalities.
Accordingly, an object of the present invention is to provide a novel method, system, and computer program product for training a massive training artificial neural network (MTANN) with a very small number of cases.
In addition, an object of the present invention is to provide a novel method, system, and computer program product for training a plurality of MTANNs (a Multi-MTANN) with a very small number of cases.
Another object of the present invention is to provide a novel method, system, and computer program product for training a MTANN to reduce false positives in the computerized detection of abnormalities in medical images.
A further object of the present invention is to provide a novel method, system, and computer program product for training a MTANN to reduce false positives in the computerized detection of lung nodules in LDCT.
These and other objects are achieved according to the invention by providing a novel method, system, and computer program product for selecting an operational set of training images for a massive training artificial neural network (MTANN), the MTANN configured to output an indication of an abnormality in a test image, comprising: (1) selecting a prospective set of training images from a set of domain images; (2) training the MTANN with the prospective set of training images; (3) applying a plurality of images from the set of domain images to the trained MTANN to obtain respective scores for the plurality of images; and (4) determining the operational set of training images based on the applied plurality of images and the respective scores.
According to another aspect of the present invention, there is provided a method, system, and computer program product for selecting a plurality of new training images comprising: (1) determining a set of abnormality images and a corresponding set of abnormality scores from the plurality of images and the respective scores; (2) selecting, from the set of abnormality images, an abnormality image having a minimal score in the corresponding set of abnormality scores; (3) determining a set of non-abnormality images and a corresponding set of non-abnormality scores from the plurality of images and the respective scores, wherein each image in the set of non-abnormality images has a corresponding score greater than the minimal score in the corresponding set of abnormality scores; (4) selecting, from the set of non-abnormality images, a non-abnormality image having a median score in the corresponding set of non-abnormality scores; and (5) selecting the abnormality image and the non-abnormality image as the plurality of new training images.
In addition, according to another embodiment of the present invention, the method of selecting a set of operational training images for a massive training artificial neural network (MTANN) further comprises: (1) calculating a performance measure of the MTANN based on the applied plurality of images and the respective scores; (2) setting the prospective set of training images to be the operational set of training images; and (3) repeating the training, applying, determining, calculating, and setting steps until the performance measure of the MTANN decreases
According to another aspect of the present invention, there is provided a method, system, and computer program product for selecting training images for a plurality of MTANNs comprising a Multi-MTANN, wherein each MTANN in the Multi-MTANN is configured to output an indication of an abnormality in the test image and the output of each of the plurality of MTANNs is combined to form a combined indication of the abnormality in the test image, the method comprising: (1) selecting a set of training images for a selected MTANN in the Multi-MTANN using the method described above; (2) training the selected MTANN with the selected set of training images; (3) activating the trained MTANN within the Multi-MTANN; (4) applying a plurality of images from the set of domain images to the Multi-MTANN to obtain respective scores; (5) selecting a second set of training images for a second selected MTANN in the Multi-MTANN based on the applied plurality of images and the respective scores; and (6) repeating the previous training, activating, applying, and selecting steps until a predetermined condition is satisfied.